[Build] Rework internal build plugin plugin to work with Isolated Projects #123461

breskeby · 2025-02-26T08:27:34Z

This fixes a general flaw in our build logic where we reach out to configurations of other projects.
This is not best practice and breaks future initiatives like IsolatedProjects that allow parallel configuration
of subprojects.

elasticsearchmachine · 2025-03-13T11:25:07Z

Pinging @elastic/es-delivery (Team:Delivery)

jozala

LGTM

rjernst · 2025-03-14T14:19:19Z

modules/aggregations/build.gradle

  classname ='org.elasticsearch.aggregations.AggregationsPlugin'
-  extendedPlugins = ['lang-painless']
+  extendedPluginProjects {
+    "lang-painless" {


This syntax seems a bit verbose. Is there a way we could use just the project path, and resolve something from the project at runtime to determine the plugin name? Most of the time this is just the project name, so having to specify it twice along with the extra syntax feels like unnecessary boilerplate.

The problem is that unfortunately the naming is not deterministic across the different plugins. At the moment the actual name is declared in the build script only. We could resolve the name maybe by resolving the generated plugin properties, but for now I prefer this little bit boilerplate in the definition over the constant cost of trying to be clever here and resolve that dynamically.

Looking at this again, I just realize we might have a better way to move forward here as we only need the the project path atm to configure a potential test cluster automatically. Given we use less and less the test cluster plugin we probably tweak this to not generally change the plugin declaration but just do the module dependency in the test clusters explicitly.

puh, let me revisit this again 🤡

@rjernst turns out alls this was basically not required in our current codebase. see https://github.com/elastic/elasticsearch/pull/123461/files#r1995753719

- Update restrited buildservice

breskeby · 2025-03-14T15:09:56Z

...-internal/src/main/java/org/elasticsearch/gradle/internal/BaseInternalPluginBuildPlugin.java

        if (isModule == false || isXPackModule) {
            addNoticeGeneration(project, extension);
        }
-        project.afterEvaluate(p -> {


Actually this was only used by internal plugins that

a) still rely on test cluster plugin,
b) extend other plugins
c) do not use default distribution

which is actually ZERO

rjernst

LGTM. FWIW it would be nice to not have to specify dependencies twice, once in the plugin properties block, and again under dependencies. But this is good on its own.

breskeby · 2025-03-14T16:48:32Z

LGTM. FWIW it would be nice to not have to specify dependencies twice, once in the plugin properties block, and again under dependencies. But this is good on its own.

true. we would need some stronger modelling here for this.
At the moment we want to put the focus first on cleaning our plugins and scripts from deprecated / discouraged api usages and accept some downsides in some verbosity for now I think.

…jects (elastic#123461) This fixes a general flaw in our build logic where we reach out to configurations of other projects. This is not best practice and breaks future initiatives like IsolatedProjects that allow parallel configuration of subprojects.

elasticsearchmachine · 2025-03-17T07:29:40Z

💚 Backport successful

Status	Branch	Result
✅	8.18
✅	8.x
✅	9.0
✅	8.16
✅	8.17

…jects (#123461) (#124993) This fixes a general flaw in our build logic where we reach out to configurations of other projects. This is not best practice and breaks future initiatives like IsolatedProjects that allow parallel configuration of subprojects.

…jects (#123461) (#124995) This fixes a general flaw in our build logic where we reach out to configurations of other projects. This is not best practice and breaks future initiatives like IsolatedProjects that allow parallel configuration of subprojects.

…jects (#123461) (#124992) This fixes a general flaw in our build logic where we reach out to configurations of other projects. This is not best practice and breaks future initiatives like IsolatedProjects that allow parallel configuration of subprojects.

…jects (#123461) (#124991) This fixes a general flaw in our build logic where we reach out to configurations of other projects. This is not best practice and breaks future initiatives like IsolatedProjects that allow parallel configuration of subprojects.

* main: (95 commits) Mute org.elasticsearch.datastreams.lifecycle.DataStreamLifecycleServiceIT testLifecycleAppliedToFailureStore elastic#124999 Merge template mappings properly during validation (elastic#124784) [Build] Rework internal build plugin plugin to work with Isolated Projects (elastic#123461) [Build] Require reason for usesDefaultDistribution (elastic#124707) Mute org.elasticsearch.packaging.test.DockerTests test011SecurityEnabledStatus elastic#124990 Mute org.elasticsearch.xpack.ilm.TimeSeriesDataStreamsIT testRolloverAction elastic#124987 Mute org.elasticsearch.packaging.test.BootstrapCheckTests test10Install elastic#124957 Mute org.elasticsearch.integration.DataStreamLifecycleServiceRuntimeSecurityIT testRolloverLifecycleAndForceMergeAuthorized elastic#124978 Mute org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryStopIT testStopQuery elastic#124977 Mute org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryStopIT testStopQueryLocal elastic#121672 Mention zero-window state in networking docs (elastic#124967) Remove remoteAddress field from TransportResponse (elastic#120016) Include failures in partial response (elastic#124929) Prevent work starvation bug if using scaling EsThreadPoolExecutor with core pool size = 0 (elastic#124732) Re-enable analysis stemmer test (elastic#124961) Mute org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryStopIT testStopQueryLocalNoRemotes elastic#124959 ESQL: Catch parsing exception (elastic#124958) ESQL: Improve error message for ( and [ (elastic#124177) Mute org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT test {lookup-join.MvJoinKeyFromRow SYNC} elastic#124951 Mute org.elasticsearch.datastreams.lifecycle.DataStreamLifecycleServiceIT testErrorRecordingOnRetention elastic#124950 ... # Conflicts: # server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java # server/src/test/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldTypeTests.java

…jects (elastic#123461) This fixes a general flaw in our build logic where we reach out to configurations of other projects. This is not best practice and breaks future initiatives like IsolatedProjects that allow parallel configuration of subprojects.

breskeby added >non-issue :Delivery/Build Build or test infrastructure Team:Delivery Meta label for Delivery team auto-backport Automatically create backport pull requests when merged v8.18.1 v8.19.0 v9.0.1 v9.1.0 v8.16.6 v8.17.4 labels Feb 26, 2025

breskeby self-assigned this Feb 26, 2025

breskeby force-pushed the fix-extended-plugin-lookup branch from c79bac8 to ca0117d Compare February 26, 2025 11:55

breskeby force-pushed the fix-extended-plugin-lookup branch from ed1832e to 9759f49 Compare March 13, 2025 09:49

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Mar 13, 2025

breskeby marked this pull request as ready for review March 13, 2025 11:24

breskeby requested review from a team as code owners March 13, 2025 11:24

jozala approved these changes Mar 14, 2025

View reviewed changes

rjernst reviewed Mar 14, 2025

View reviewed changes

declare test cluster module requirement explicitly in mapper-murmur

dfe7d45

- Update restrited buildservice

breskeby force-pushed the fix-extended-plugin-lookup branch from 484b12a to dfe7d45 Compare March 14, 2025 15:02

do not autoconfigure test cluster modules for internal plugin extensions

43fb029

breskeby commented Mar 14, 2025

View reviewed changes

Murmur3 plugin not using test cluster

ce2aef6

breskeby force-pushed the fix-extended-plugin-lookup branch from c3ab3e4 to ce2aef6 Compare March 14, 2025 15:13

breskeby requested a review from rjernst March 14, 2025 15:19

[CI] Auto commit changes from spotless

78a4fa1

rjernst approved these changes Mar 14, 2025

View reviewed changes

Merge branch 'main' into fix-extended-plugin-lookup

46a33f4

Merge branch 'main' into fix-extended-plugin-lookup

3753357

breskeby merged commit 7f722e9 into elastic:main Mar 17, 2025
14 of 17 checks passed

breskeby mentioned this pull request Mar 17, 2025

[8.16] [Build] Rework internal build plugin plugin to work with Isolated Projects (#123461) #124994

Closed

breskeby mentioned this pull request Mar 17, 2025

[8.17] [Build] Rework internal build plugin plugin to work with Isolated Projects (#123461) #124995

Merged

breskeby deleted the fix-extended-plugin-lookup branch March 17, 2025 07:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Build] Rework internal build plugin plugin to work with Isolated Projects #123461

[Build] Rework internal build plugin plugin to work with Isolated Projects #123461

Uh oh!

breskeby commented Feb 26, 2025

Uh oh!

elasticsearchmachine commented Mar 13, 2025

Uh oh!

jozala left a comment

Uh oh!

rjernst Mar 14, 2025

Uh oh!

breskeby Mar 14, 2025

Uh oh!

breskeby Mar 14, 2025

Uh oh!

breskeby Mar 14, 2025 •

edited

Loading

Uh oh!

rjernst left a comment

Uh oh!

breskeby commented Mar 14, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[Build] Rework internal build plugin plugin to work with Isolated Projects #123461

[Build] Rework internal build plugin plugin to work with Isolated Projects #123461

Uh oh!

Conversation

breskeby commented Feb 26, 2025

Uh oh!

elasticsearchmachine commented Mar 13, 2025

Uh oh!

jozala left a comment

Choose a reason for hiding this comment

Uh oh!

rjernst Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

breskeby Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

breskeby Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

breskeby Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rjernst left a comment

Choose a reason for hiding this comment

Uh oh!

breskeby commented Mar 14, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 17, 2025

💚 Backport successful

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

breskeby Mar 14, 2025 •

edited

Loading